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The ability to reason scientifically and communicate research appropriately is central to psychological literacy. 
Scientific research has little value unless scientists are able to convey results and their consequences clearly 
to others. In this study, we outline a method of assessing the development of psychological literacy in 
undergraduate students. Data from three cohorts undertaking assessed intervieius as part of a final year 
research project unit are examined. This assessment evaluated students ’ ability to explain the purposes and 
findings of their research to a lay audience, to articulate the conceptual basis and methodological background 
to their approach, and to reflect on their oiun development during the research process. Analysis of marks and 
feedback from the assessment suggests it provides a reliable, means of evaluating skills that contribute to 
psychological literacy in a manner that, is both, educationally acceptable to students and a valid, indicator of 
their general, level of course performance. 
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Introduction 

HIS STUDY presents and evaluates the 
use of oral examinations as a means of 
assessing undergraduate psychology stu¬ 
dent learning during their final year empir¬ 
ical research project. Learning how to 
communicate scientific findings to others in 
speech as well as writing, is an important part 
of the development of undergraduate psy¬ 
chologists which can contribute to their 
future employability. In this study, we 
examine the reliability and validity of this 
form of assessment and consider some of the 
educational and practical implications of 
using oral examinations to assess psycho¬ 
logical literacy. 

Psychology remains one of the most pop¬ 
ular degree choices with over 100,000 appli¬ 
cations made annually to study the subject at 
UK universities (Universities and Colleges 
Admissions Service [UCAS], 2015). The con¬ 
tinued expansion of higher education, to 
which the subject area of psychology con¬ 
tributes, is founded on the premise that uni¬ 
versity study will reward students in the long 
term through the opportunity to progress 
into better paid, highly skilled employment 


(Morrison, 2014). Growth in participation 
rates has placed an increased expectation on 
higher education institutions to supply grad¬ 
uates who are ready for the demands of the 
modern workplace and possess the profes¬ 
sional skills that employers want (e.g. 
Docherty & Fernandez, 2014; Tomlinson, 
2010). An additional challenge for the psy¬ 
chology subject area is that its graduates are 
known to take longer to progress into grad¬ 
uate careers following university and less 
than 20 per cent are thought to eventually 
become professional psychologists (e.g. 
Trapp et al., 2011; Van Laar & Udell, 2008). 
In a survey of four cohorts of psychology 
graduates between one and seven years post¬ 
graduation, Coulthard (2013, 2015) found 
that only 40 per cent of psychology students 
were in full-time employment 12 months 
after graduating increasing to 67 per cent 
four years after graduation. Moreover, only 
60 per cent of psychology graduates reported 
that having a degree in psychology had been 
necessary for obtaining their current employ¬ 
ment, confirming that a large proportion of 
psychology graduates may not explicitly make 
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use of subject knowledge from their degree 
following graduation (Lantz, 2011). It is, 
therefore, a growing concern within higher 
education to provide opportunities for 
students to gain work-related skills and expe¬ 
rience during their degree (Quality Assur¬ 
ance Agency [QAA], 2014); and for 
psychology students in particular, to incorpo¬ 
rate professional development as a core stan¬ 
dard within undergraduate curricula (British 
Psychological Society [BPS], 2014). 

One approach important to the develop¬ 
ment of the employment potential of psy¬ 
chology students is the concept of 
‘psychological literacy’. Psychological lit¬ 
eracy is a broad concept that is assumed to 
encapsulate employability skills, global citi¬ 
zenship and scientific understanding 
(Cranney et al., 2013). By emphasising to 
students the scientific, evidence-based rea¬ 
soning implicit within the study of psy¬ 
chology and its value and relevance to 
different professions, it is possible that the 
employability of graduates can be enhanced 
(Mair, Taylor & Hulme, 2013). Since psycho¬ 
logical understanding might be used benefi¬ 
cially by graduates to help others function 
more effectively and ethically within the 
workplace or wider society, it is necessary for 
providers of psychology programmes to 
present students with opportunities to ‘apply 
their skills and knowledge to authentic prob¬ 
lems in a range of contexts that demonstrate 
the broad application of psychological 
theory to real life and work situations.’ (Mair 
et ah, 2013, p.6). The challenge faced by 
education providers then is to consider how 
psychological literacy might be better inte¬ 
grated into undergraduate degree pro¬ 
grammes. 

Several distinct but related interpreta¬ 
tions of psychological literacy exist (e.g. 
Cranney et al., 2011; McGovern et al., 2010; 
Trapp et al., 2011) which indicate the con¬ 
cept to be both broad and complex (Roberts, 
Heritage & Gasson, 2015). Central to most 
definitions lie four key principles whereby 
the demonstration of psychological literacy 
should include: (i) fluency in the core knowl¬ 


edge, concepts and practice of psychology; 
(ii) use of scientific inquiry and critical 
thinking; (iii) ethical and socially responsible 
actions and behaviour; and (iv) professional 
awareness, development and reflection. 

A further pervasive theme evident in 
most definitions of psychological literacy 
and implicit within the sub-principles is an 
overarching requirement for effective 
communication. Whether presenting or 
reporting research findings, communicating 
psychological beliefs to the public, or inter¬ 
acting successfully with others, someone who 
is ‘psychologically literate’ should be capable 
of expressing relevant information in a 
manner appropriate to their intended audi¬ 
ence. As Trapp et al. (2011) have indicated, 
central to the notion is communicating the 
value of psychology in everyday contexts, 
such that when engaging with communities, 
employers and the media the real-life appli¬ 
cability of psychological skills and knowledge 
can be conveyed and common mispercep¬ 
tions about psychology can be challenged. 
Since many problems within contemporary 
society (e.g. obesity, work-life balance, radi- 
calisation) are behaviourally based, the more 
those trained in psychology are able to com¬ 
municate their understanding to others, the 
greater the net benefit to society may poten¬ 
tially be (Cranney, Botwood & Morris, 2012). 

Whilst there is common agreement that 
psychological literacy should be a primary 
outcome of an undergraduate psychology 
education both in the UK (Mair et al., 2013) 
and elsewhere (e.g. Cranney et al., 2012; 
Mnerican Psychological Association [APA], 
2013), the abstract and multi-dimensional 
nature of the construct presents a barrier to 
its integration into curricula with some 
authors suggesting there is an urgent need 
to more clearly specify an operational defini¬ 
tion of the concept, its boundaries and how 
it can be measured in order to improve its 
real-world utility (Roberts et al., 2015). A 
Halpern and Butler (2011) argue, simply 
calling for students to be educated about 
psychological literacy may prove hollow 
unless an effective means of assessing how 
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well students develop this collection of 
attributes is developed. Assessments that 
evaluate psychological literacy are, there¬ 
fore, required. 

Some attempts at examining psycholog¬ 
ical literacy in undergraduates have 
explored its links to more generic graduate 
skills and abilities. Morris et al. (2013) evalu¬ 
ated a cross-section of undergraduates, 
finding that whilst the importance of psy¬ 
chological literacy was rated highly by 
students, the extent to which students felt 
they had developed these skills during their 
studies was rated only moderately. Using 
psychometric test responses from over 500 
students in an Australian university, Roberts 
et al. (2015) concluded that psychological 
literacy could best be characterised as con¬ 
sisting of three primary components: reflec¬ 
tive processes, general academic attributes, 
and the perception of psychology as a 
helping profession, although scores on these 
dimensions did not correlate strongly with 
students’ self-rated assessment of nine indi¬ 
vidual characteristics derived from the defi¬ 
nition of psychological literacy proposed by 
McGovern et al. (2010). The authors con¬ 
clude that further objective means of identi¬ 
fying psychological literacy are required. 

Potential activities that might provide a 
means of embedding psychological literacy 
within undergraduate curricula have been 
considered by some authors. For example, 
Mair et al. (2013) identify several concepts 
common to undergraduate psychology 
syllabuses (e.g. social influence, mental 
health, resilience) where students might be 
able to use their knowledge to benefit 
others, whilst Taylor and Mair (2013) suggest 
three different ways of encouraging students 
to reflect on social psychological aspects of 
their learning including tasks exploring envi¬ 
ronmental behaviour, teamwork-based activi¬ 
ties and mock interviews. Trapp et al. (2011) 
proposed setting applied problems within 
businesses or the local community such that 
employers and students are able to see how 
taught materials have real-world relevance. 
The authors also recommend that under¬ 


graduate curricula should require a diverse 
range of assessment practices so that all skills 
defined in the psychology benchmark state¬ 
ment (QAA, 2010) are evaluated. Moreover, 
to improve the employability of graduates, 
students should be encouraged to communi¬ 
cate the skills they develop during their 
degree to others, rather than assuming the 
abilities that psychology graduates possess 
will be publically recognised. It follows that 
the development of assessments that pro¬ 
mote an evaluation of the multi-dimensional 
components of psychological literacy and 
also allow students to practice the communi¬ 
cation and demonstration of these attributes 
will be key to successfully embedding and 
enhancing its inclusion within undergrad¬ 
uate curricula. 

Several authors have suggested scenario- 
based approaches to assessment whereby 
students are asked to adaptively apply psy¬ 
chological concepts to explain events or pro¬ 
pose solutions. Such activities might include 
evaluating newspaper editorials or advertise¬ 
ment claims (e.g. Halpern & Butler, 2011), 
writing letters intended for a non-psycholog- 
ical audience outlining the evidence for and 
against a particular course of action (e.g. 
Cranney et al., 2013) or case study analyses 
which require students to implement a 
strategy or propose an intervention (e.g. 
McGovern et al., 2010). A common issue 
with situated learning experiences, however, 
is that they can be difficult for students to 
accomplish in a limited time frame without 
over trivialising the nature of the task. This 
makes the standardisation of such tasks for 
the purpose of assessment challenging given 
the ambiguity which exists in the potential 
approaches to each topic and the variable 
outcomes that may be obtained. Cranney 
et al. (2013) advocate a portfolio-based 
approach, where students are required to 
build up evidence and reflect on their skills 
development across several modules on their 
degree, or the use of ‘capstone’ modules 
which require students to apply the knowl¬ 
edge and skills learned from earlier in their 
studies. The nature of capstone units, 
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however, is known to vary widely across insti¬ 
tutions and may be taken to include intern¬ 
ships, research projects, outreach 
experiences or a mixture of these options 
(Weimer, 2012). Therefore, the need exists 
for more consistent methods of assessing psy¬ 
chological literacy to be developed which 
might readily fit into undergraduate UK 
degree programmes and which are relevant 
to the diverse range of attributes which con¬ 
tribute to the concept. 

One possibility for the assessment of psy¬ 
chological literacy which also satisfies the 
related goal of contributing to the develop¬ 
ment of employability skills in students is the 
use of oral ( viva voce) examinations. Using 
oral examinations to assess student compe¬ 
tencies is well established in many subject 
areas including medicine (e.g. Evans, Inger- 
soll & Smith, 1966) and law (e.g. Butler & 
Wiseman, 1993). Viva voce examinations also 
have a long tradition in the assessment of 
doctoral research and have been shown to 
increase selfperceptions of academic com¬ 
petence in PhD students (Jackson & Tinkler, 
2001 ). 

The pedagogic benefits of interview- 
based oral examinations have been well 
researched. In most forms of the assessment, 
a set of questions are developed that cover 
core aspects of domain knowledge and skills 
that students are expected to demonstrate. 
Both the students’ depth of comprehension 
of this content and their ability to communi¬ 
cate this effectively can, therefore, be evalu¬ 
ated (Joughin, 1998). The approach allows 
examiners to explore topics in direct conver¬ 
sation with students such that knowledge can 
be further interrogated and questions can be 
clarified, maximising opportunities for 
students to demonstrate their full potential. 
Students are forced to rely on their own 
words and understanding strengthening the 
academic integrity of the assessment and 
reducing the potential for plagiarism. Oral 
assessment is also thought to improve depth 
of learning with some evidence suggesting 
that students prepare more thoroughly for 
this form of assessment to help improve con¬ 


fidence in their ability to deal with questions 
and to avoid feeling foolish in front of 
the examiner (Butler & Wiseman, 1993; 
Joughin, 2003). The presence of examiner 
panels with whom the student interacts 
directly, whilst providing motivation, can also 
lead to stress in students and may impede 
their ability to perform to their best, 
although research evidence that might con¬ 
firm a direct link between anxiety levels and 
students’ oral performance in viva voce 
examinations appears inconclusive (Arndt, 
Guly & McManus, 1986). A further factor to 
consider in the use of oral examinations is 
the unavoidable lack of anonymity for the 
student, which may contravene assessment 
protocols in some universities. As a conse¬ 
quence, assessments may be influenced by 
prior knowledge of the student or be subject 
to other sources of bias similar to those 
found in selection interviews (e.g. Arvey, 
1979). To protect against this, safeguards are 
required to standardise the interview 
process, assessment criteria used, and alloca¬ 
tion of examiners to students. 

Oral examinations have been used suc¬ 
cessfully to evaluate business communication 
skills in human resource undergraduates 
(Burke-Smalley, 2014), assess pharmaceutical 
students’ confidence in care settings (Sib- 
bald, 1998) and evaluate understanding of 
customer service relationships in marketing 
undergraduates (Pearce & Lee, 2009). How¬ 
ever, evidence regarding the extent to which 
oral assessments are indicative of a student’s 
overall level of ability appears mixed. Oakley 
and Hencken (2005) used 30-minute 
assessed interviews with undergraduate 
sports science students, finding performance 
on the assessment where six interview ques¬ 
tions were drawn at random correlated posi¬ 
tively with end-of-year exam scores. Torke et 
al. (2010) compared the performance of 
medical students on a written theory exami¬ 
nation with their performance during a 10- to 
15-minute viva voce assessment that con¬ 
tributed to the same module. Students were 
given a ‘viva card’ of preliminary interview 
topics whilst waiting their turn to take the 
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examination in an attempt to allow students 
to mentally prepare, although examiners 
deviated to other topics later during the 
examination. The authors found that whilst 
the ratio of students passing and failing each 
assessment did not vary, overall scores on the 
two assessments did not correlate well, con¬ 
cluding that whilst viva voce examinations 
may be suitable to differentiate between top 
performing students where more in-depth 
questions can be posed, they should not be 
used in isolation to determine whether a stu¬ 
dent passes or fails a subject. 

An old but comprehensive study con¬ 
ducted by Evans et al. (1966) similarly found 
no correlation between 20-minute oral 
examinations and an objective written exam¬ 
ination taken two days later by students on 
the same module. In this study, medical 
students were interviewed twice by different 
pairs of assessors. Good agreement (r>0.7) 
was found between junior and senior physi¬ 
cians within each interview team, and 
between different teams of assessors. Oral 
examination grades also correlated positively 
with the total amount of time each student 
spent talking and the number of words 
spoken during the interview which the 
authors attributed to more fluent speakers 
being better able to organise their thoughts. 
However, no relationship was found between 
spoken interview performance and grading 
of the same interviews from verbatim written 
transcripts, when marked independently two 
months later; described by the authors as 
‘oral parallax’ (p.654) given the shift in eval¬ 
uation of the same information when con¬ 
sidered via a different medium. More 
recently, Huxham, Campbell and Westwood 
(2012) also found undergraduate biology 
students scored significantly higher when 
assessed by oral examination compared to 
students who answered the same questions 
by written examination, which the authors 
suggest may be related to an increased sense 
of professionalism in the oral context. This 
pattern held for questions that required 
scientific analysis and those which asked 
students to reflect on skills development. 


The existing literature shows that oral 
examinations are used across different aca¬ 
demic disciplines to improve student skills 
where the assessment format has clear voca¬ 
tional relevance, although evidence 
regarding the predictive validity of oral 
examinations as an indicator of general 
course performance is less clear cut, and 
may be dependent on the interview method¬ 
ology used. In addition, relatively little is 
known about the utility of oral examinations 
on undergraduate courses within the psy¬ 
chology subject area, despite the obvious 
importance of oral communication skills 
within the profession. The use of interviews 
to assess learning from undergraduate 
research projects not only fits well with the 
core aspects of the psychology subject bench¬ 
mark (QAA, 2010) but also encourages 
students to engage in social conversation 
about the validity of the scientific conclu¬ 
sions, a central component of psychological 
literacy (Cranney et al., 2013). Whilst most 
scientists consider the reporting and visi¬ 
bility of their research in broadcast and print 
media to be important (Peters, 2013) this 
process can often lead to the misrepresenta¬ 
tion of findings through over-simplification, 
exaggeration or omission of critical detail, 
such that some researchers feel ill-prepared 
in the art of public communication (Kaye et 
al., 2011; Wien 2014). Misconceptions about 
psychology, outside of the scientific disci¬ 
pline, are common in everyday society and it 
is important for psychologists to help 
develop public understanding by chal¬ 
lenging incorrect assumptions. It follows that 
engaging undergraduates with opportunities 
to explain research findings in a manner 
that is appropriate to different audiences 
should be a fundamental goal of undergrad¬ 
uate programmes in psychology. The ability 
to present scientific arguments and commu¬ 
nicate findings accurately will be key to 
improving the scientific literacy of the 
general public, and may help counter public 
scepticism about psychology and its ability to 
address significant issues within society. As 
Crowe (2012, p.58) points out, ‘Psychology 
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leaders, educators, and graduates should be 
capable of “giving psychology away” to recep¬ 
tive members of the public, many of whom 
will have influence on their immediate and 
wider communities.’ 

The current study, therefore, seeks to 
investigate the use and effectiveness of oral 
examinations as a means of assessing the psy¬ 
chological literacy of undergraduate 
students by examining their application to a 
substantial core research component of all 
BPS accredited courses, the final year 
research project. The specific aims of the 
study, were: 

• To examine the validity of assessed oral 
project interviews to predict students’ 
performance elsewhere on their psy¬ 
chology degree. 

• To examine the reliability of oral project 
interviews as an assessment method 
where different teams of interviewers are 
used. 

• To examine students’ views on the 
acceptability and educational impact of 
oral project interviews. 

• To consider the impact of prior mock 
interviews on students’ preparedness and 
perceptions of assessed interviews. 

Method 

Sample 

Participants were 454 final year undergrad¬ 
uate students enrolled on two BPS accred¬ 
ited undergraduate psychology programmes 
at one university within the UK. As part of 
these degree programmes, all students com¬ 
plete a compulsory 40-credit empirical 
research project unit, equivalent to 33 per 
cent of the total marks available during their 
final year. 

Data from three cohorts of students are 
examined of whom 443 attended an assessed 
project interview as part of the final year 
project unit. The remaining 11 students did 
not complete the assessment on grounds of 
reasonable adjustment, extenuating circum¬ 
stances on the day of the interview or non¬ 
completion of the academic year. The final 
data sample, therefore, comprised those 


graduating in 2013 (A^163), 2014 (AM55) 
and 2015 (Af=125), of whom 72 (16 per cent) 
were male and 371 (84 per cent) were 
female. 

Project interview design and preparation 
The project interview was designed to meet 
three key objectives. Firstly, to allow students 
to demonstrate their ability to explain 
aspects of their final year project as if to an 
interview panel unfamiliar with their work 
and with only a general awareness of the dis¬ 
cipline. Secondly, to give students an oppor¬ 
tunity to expand on the written account of 
their project with respect to the process of 
conducting research and identifying 
learning from this culminating aspect of 
their degree. Thirdly, to assess the extent to 
which students’ interview performance was 
consistent with the standard of scientific rea¬ 
soning and communication skills expected 
of a graduate of psychology. 

Project interviews lasted for 15 minutes. 
All interviews were timed so as not to extend 
beyond this limit and audio recorded for the 
purposes of later mark verification. Students 
were advised that whilst interviewers would be 
friendly and try to place them at ease, the con¬ 
duct of the interview was a formal summative 
assessment and they should respond accord¬ 
ingly. Interviews started and finished at a des¬ 
ignated time, such that late arrivals would 
have time deducted from their interview. 

The interview assessment contributed 
10 per cent of the student’s overall mark for 
the project unit, with the remaining 90 per 
cent being derived from a written project 
report of up to 8000 words. This weighting 
was chosen to strike a balance between lim¬ 
iting student anxiety arising from the credit 
value of the interview whilst still providing 
students with an opportunity to improve 
upon the overall degree class of their 
project. The assessment regulations of our 
institution also meant that a student could 
not successfully pass the project unit without 
attempting all assessments on the unit. Inter¬ 
views were held approximately seven weeks 
after students had completed and submitted 
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their written project reports. However, to 
limit the diverse effects that a good or bad 
report mark may have on student anxiety or 
motivation during the interview, the decision 
was made not to release report marks to 
students prior to interview. 

Students were informed prior to the 
interviews that they should address their 
answers as if to an intelligent lay person or 
professional who was not familiar with the 
specifics of their study. Project supervisors 
were not included on interview panels, such 
that interviewers were naive to the nature of 
each student project. Students were also 
informed that they could bring a copy of 
their project report to the interview if they 
wished, but that this was not necessary and 
not advised. It was not the purpose of the 
interview to examine the detailed informa¬ 
tion in their work, but their ability to explain 
their work that was of primary importance. 
For this reason, interviewers read only the 
abstract of the student’s work prior to each 
interview, to familiarise themselves with the 
basic nature of the study conducted. 

As Oakley and Hencken (2005) recom¬ 
mend, student anxiety can be reduced by 
making students more familiar with the 
structure and style of the assessment before¬ 
hand. To help students prepare for the inter¬ 
view, online tutorial materials were 
developed consisting of a guide to the inter¬ 
view process, three videos of full interviews 
conducted with ex-students showing good 
and bad answers to different questions, a 
pool of practice interview questions, and an 
online forum to which students could post 
questions. Additionally, a special project 
interview workshop was held six weeks prior 
to the assessment period where marking 
criteria were discussed and any further 
queries could be addressed. Students were 
also given the opportunity to practice 
responding to interview questions via a mock 
interview with their project supervisor after 
completion of their project report. Super¬ 
visors were asked to make use of the practice 
questions available within the online tutorial 
for this purpose. 


Operational procedures 

Interviews were held over a three-day period 
two weeks prior to the students’ end-of-year 
examinations. All interviews were held in the 
same location comprising six adjacent 
research rooms within the psychology depart¬ 
ment, with students first being asked to report 
to a central waiting area from where they were 
collected. To help standardise practice 
between different interviewers, all interviewers 
attended a group training session covering 
expected interview questioning and marking 
protocol prior to the main interviews. 

Each student was assessed by a panel of 
two academic staff. The first interviewer 
acted as the lead interviewer, covering set 
themes from a pre-defined list of compulsory 
questions such that all candidates were asked 
the same core questions. The second inter¬ 
viewer acted as the primary marker, making 
notes about the candidate’s answers and 
scoring their responses in each question cat¬ 
egory as the interview progressed. If time 
permitted and where relevant to the candi¬ 
date’s earlier responses, the second inter¬ 
viewer would also ask follow-up questions 
selected from a supplementary list of ques¬ 
tions, used by all interview teams. Inter¬ 
viewers were permitted to be encouraging 
and supportive by repeating or rephrasing 
questions if required, but were not per¬ 
mitted to assist students beyond this. 

Depending on cohort size, five or six 
interview teams were used to assess all 
students, such that each team undertook 
between 25 to 30 interviews over a three-day 
period. First and second interviewers were 
systematically rotated over the three-day 
assessment period such that each first inter¬ 
viewer marked with each second interviewer. 
The reliability of oral assessments has been 
shown to increase when multiple examiners 
are used (Wass et al., 2003). This arises since 
sharing perceptions helps interviewers 
become more aware of the inferences they 
make, information is less likely to be missed, 
and bias in decision-making is reduced since 
interviewers provide checks on each other 
(Campion, Palmer & Campion, 1997). 
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Students were allocated to scheduled 
interview slots which exceeded the expected 
duration of the interview so that the inter¬ 
view procedure could be explained to the 
student, to allow for overrun, and to provide 
time for the interviewers to agree marks and 
complete a feedback sheet with comments 
for the student, before starting their next 
interview. Assessment and feedback was 
therefore completed instantaneously, with 
both project report and interview results 
being made available to all students two 
working days following the date of the last 
interview sessions. 

Interview questions and marking criteria 
Students were asked questions structured 
around five main themes. The five themes 
required students to: (i) give a concise non¬ 
specialist explanation of their project; (ii) 
explain the rationale for their study; (iii) 
demonstrate an understanding of method¬ 
ological issues concerning their study; (iv) 
provide an interpretation of their findings; 
and (v) reflect on what they had learned 
from the project process. Themes were 
chosen to reflect the nature of questions that 
a recent graduate might reasonably be 
expected to answer about their research 
project when attending a selection interview 
following university. 

Each theme consisted of at least two com¬ 
pulsory questions asked by the lead inter¬ 
viewer and a further three supplementary 
questions that could be asked by the second 
interviewer. Using the same question pool 
and asking questions in the same order 
whilst limiting requests for elaboration is 
known to improve the consistency of the 
interview process (Campion et al., 1997). 
However, some variety in follow-up questions 
was felt desirable to help reduce the possi¬ 
bility of questions being passed between 
students tested on different days (Oakley & 
Hencken, 2005). Interviewers attempted to 
devote around the same amount of time to 
each theme, with the second interviewer 
monitoring the elapsed time and moving the 
discussion on, as required. 


To facilitate the opening of the interview, 
the first two questions were disclosed to 
students before the assessment. These were: 
‘How would you explain your project to a non-psy¬ 
chologist'? ’ and How did the idea for your project 
emerge? ’Student responses to each of the five 
interview themes were graded by the second 
interviewer on a discontinuous percentage 
scale, with a mark being awarded for each 
theme. An overall mark was then calculated 
by the second interviewer, determined as the 
mean score of these five themes plus an addi¬ 
tional mark based on their global assessment 
of the candidate’s performance during the 
interview. 

The lead interviewer, blind to the marks 
awarded by the second interviewer, would 
also independently provide an overall mark 
for the interview. A final interview mark was 
then agreed following discussion between 
the two interviewers, taking into considera¬ 
tion the first and second interviewer marks 
and interview marking criteria. Marking 
criteria in the first class category for each 
interview theme plus the global assessment 
of the candidate are shown in Table 1. 

Results 

Concurrent validity 

To examine project interviews in the context 
of students’ performance elsewhere on their 
degree, agreed interview marks were com¬ 
pared with project report marks, final year 
course averages and students’ overall degree 
classification upon graduation. Paired- 
samples t-tests were used to examine whether 
interview and project report grades differed 
for each student. When all cohorts were 
aggregated, no significant difference was 
found between interview marks (M= 67.2%; 
SD= 8.9%) and project report marks 
(M= 66.6%; SD=7.1%), t( 437 df)= 1.27, p=. 21 
n.s., d= 0.07. This was also true when inter¬ 
view and project marks were examined 
separately for each cohort (Table 2). 

Significant positive correlations, with a 
moderate effect size were found between 
project interview and project report marks 
(n P ). In addition, strong positive correlations 
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Table 1: Marking criteria for the final year project interview (five main interview themes plus global candidate assessment). 


Mark Turner & Marina Davila-Ross 



56 


Psychology Teaching Revieiv Vol. 21 No. 2, Autumn 2015 









Table 1: Continued. 



CM 

LO 


U QJ 

E sq 


cti -a 

. r to 
’ —r 


CO cu 

c E 


Assessing psychological literacy via project interviews 


o 

CJ 


OJ 

■a 


oj 

<u 


cu 

E 

CD 

05 

CU 

s— 

aj 

> 

cu 

i_ 

cu 
c u 
> 


cu 

E 


o 

Q. 

aj 


CJ 

"o' 

Q. 


CU 

E 

<: 

aj 

£ 

<u 


CJ 

_aj 

’o' 

Q. 


CU 

a. 

E 

o 

CJ 


_o 

cu 


Validity Coefficient 


$ 

00 

lo 

$ 

o 

LO 

$ 

05 

C 

% 

00 

CO 

$ 

o 

% 

CO 

Interview vs. 

Report 

(paired- 

samples t) 

-0.96 n.s. 

1.70 n.s. 

1.53 n.s. 








* 

* 

O' 

4_i 4_, 


05 

CD 

*— 



CD 

CD 

CO 

‘S’ & 


;? 

# 

# 

Q_ CC 


LO 

r— 

CO 



CD 


CD 



CD 

CD 

CD 








* 

£ 

* 



CD 

<— 

CO 



CO 

CO 

05 


sz 

cu 

?? 

f? 




00 

CO 

1^ 



LO 

CO 


3 


CD 

CD 

CD 

o tu 





> 





£ £ 





c 





— 

•- -s 

# 

# 

# 



LO 

LO 

LO 



05 

05 

05 


sz 

1 

1 

1 


cu 





<u 

r=T 

o' 

O' 



O 

LO 

CM 



CM 

LO 

''t 


<u 





05 

CO 

LO 

LO 


c 

CD 

LO 

CM 


cu 





cc 




C 





M- ° 





O -M 


CO 


LO 








o 

o 

O 

<J "O 


CM 

CM 

CM 

>- 2 





CD 






aj aj 
> > 
<u <u 


CO CO 
CU <U 


Psychology Teaching Review Vol. 21 No. 2, Autumn 2015 


57 

















Mark Turner■ & Marina Davila-Ross 


were found between interview marks and 
final year course averages ( n y ). These data 
suggest that students who fared better 
during the project interview also tended to 
submit better quality project reports, with 
the absolute difference in grades achieved by 
each student being small. Interestingly, inter¬ 
view marks were found to be a better pre¬ 
dictor of overall course performance than 
project report performance, with the value 
of riy (.52) being significantly larger than 
value of n P (.38) when calculated across all 
three cohorts (7^2.35, p=. 018, 2-tailed). 

Since degree classifications are not solely 
derived from final year average marks within 
our institution, further analysis was con¬ 
ducted to examine the association between 
interview performance and final degree 
class. For the three cohorts combined, 127 
students (29 per cent) achieved a first class 
interview grade, 229 (53 per cent) achieved 
an upper second class interview grade, 72 
(17 per cent) achieved a lower second class 
interview grade and four (one per cent) 
were awarded third class or fail grades. Cross 
tabulation of interview class against final 
degree class showed a significant association 
X 2 (9, A=431)=90.32, /K.001, Cramer’s 

V=.264, such that for 59 per cent of all 
students their interview class correctly pre¬ 
dicted their final degree class. 

Reliability 

Interviewer reliability was considered by 
examining levels of agreement between first 
and second interviewers, as well as grading 
differences between different interview 
teams. 

A significant difference was found in 
marks awarded by the first interviewer 
(M= 66.5%; SD= 8.3%) and second inter¬ 
viewer (M=67.1%; SD= 8.9%), i(391 df)= 3.78, 
pc. 001, although the effect size was small 
(rf=0.06). On average, second interviewers 
graded half of one per cent mark higher 
than first interviewers. When data for each 
cohort were examined separately, differ¬ 
ences between first and second interviewers 
were found in two cohorts only (Table 3). 


Examination of the relationship between 
marks awarded by first and second inter¬ 
viewers showed strong positive correlations 
(r>.94) within each cohort group, suggesting 
pairs of interviewers tended to exhibit sim¬ 
ilar grading patterns across different candi¬ 
dates (i.e. good inter-rater reliability). The 
overall consistency in marks awarded within 
the same interview team, therefore, appears 
more influential than the absolute differ¬ 
ences in marks that occurred between first 
and second interviewers. 

A further threat to reliability is possible 
bias between interview teams, whereby some 
marker pairings may grade more harshly than 
others. Agreed interview marks where there¬ 
fore examined across different marker pair¬ 
ings using independent groups analysis of 
variance (ANOVA). No significant differences 
in agreed marks were found between different 
interview teams for any of the three cohorts 
(Table 4). The difference between the most 
lenient and most severe marker pairings 
ranged from three to five per cent in each 
cohort with the mean marks for all teams 
falling within approximately 0.25 standard 
deviations of the overall cohort mean. The 
observed effect size between teams in each 
cohort were, therefore, small (q 2 ji<0.04). 

Since pairs of markers did not interview 
all students within each cohort, grading dif¬ 
ferences between interview teams will also be 
dependent on ability variations in the subset 
of students they assessed. To examine differ¬ 
ences in the grades awarded by different 
interview teams whilst controlling for the 
effect of project quality, analysis of covari¬ 
ance was used on the data from each cohort, 
with the final year project report mark as a 
covariate (Table 4). Project report marks 
were found to be a significant covariate 
i fK.OO I) of interview marks in the three 
ANCOVA analyses conducted. However, no 
significant difference in agreed interview 
marks between different marking teams was 
found for 2013 graduates T(5,150)=1.27, 
p =.281 n.s., 2014 graduates _F(5,145)=0.59, 
p =.706 n.s. or 2015 graduates T(4,119)=0.29, 
p =.883 n.s, even when the effect of project 
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Table 3: Comparison of marking agreement and inter-rater reliability between first and second interviewers. 
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report mark was taken into account. Overall, 
these data suggest no systematic differences 
occurred in the judgements made by dif¬ 
ferent marking teams in the evaluation of 
project interview performance. 

Threats to reliability might also arise 
through contrast errors, if markers become 
more lenient or severe over time through 
judgements being influenced by impressions 
of earlier candidates, or if question sets are 
compromised following the initial inter¬ 
views. To examine marking variation over 
time, marks awarded to students interviewed 
on different days were compared using inde¬ 
pendent groups ANOVA (Table 5). 

A slight trend for mark inflation of 
between one to three per cent was observed 
from day one to day three of the interviewing 
schedule, but this was not statistically signifi¬ 
cant in any of the three cohorts. In addition, 
when project report mark was introduced as 
a covariate in the analyses to compensate for 
differences in project quality, no significant 
differences were observed across the three 
testing days, with small effect sizes being 
observed (r| 2 /k 0.04) . From this analysis, 
there is, therefore, no evidence to suggest 
that assessment standards changed over time 
or that students gained an advantage by 
being tested on later days. 

Student feedback on project interviews 
The educational impact of the interviews was 
evaluated through student satisfaction 
ratings gathered as part of course feedback 
for the 2013 and 2014 cohorts. Students 
were assured that their responses would be 
treated confidentially and would only be 
used to inform the future development of 
the interview assessment. Feedback sug¬ 
gested 94 per cent of students had accessed 
the online materials regarding the project 
interview and 81 per cent had discussed the 
interview with their supervisor. Most students 
had also taken the opportunity to practice 
being interviewed, with 68 per cent 
reporting they had arranged a mock inter¬ 
view with their project supervisor, of whom 
93 per cent reported finding this helpful. 


Acceptance of the interview format was also 
high, with 97 per cent of students agreeing 
that the assessments used were appropriate; 
the perceived appropriateness of assessment 
also correlated positively ( r=.55 ) with 
students’ overall satisfaction with the project 
unit. 

Since course satisfaction data are gath¬ 
ered anonymously within our institution, it 
was not possible to relate the feedback 
shown in Table 6 to student performance 
during interviews. A further voluntary survey 
was therefore undertaken with the 2015 
cohort after graduation in which more spe¬ 
cific feedback was sought (Table 7). Whilst 
the response rate was low, no significant dif¬ 
ference was found in interview marks 
between students who did (M=71.9%, 
ST>=11.3%) and did not (M=67.1%, 
STM.7%) respond to the survey 
(f(123)=1.78, n.s.) suggesting differences in 
interview outcomes were not a source of 
non-response bias. Of those replying, 60 per 
cent reported taking part in the interview 
had been a positive experience. Moreover, 
80 per cent reported that they now felt more 
confident being able to communicate psy¬ 
chological findings to others whilst over 60 
per cent felt taking part in the assessment 
had benefited their interview skills or would 
help them when applying for jobs in the 
future. 

Of those who took part in a mock inter¬ 
view, 80 per cent agreed this had helped 
them prepare for the real assessment. Com¬ 
parison of actual interview marks between 
those who did (M= 73.2%, £TM2.1%) and 
did not (M=67.0%, SD=6.2%) report having 
a mock interview suggested a trend for mock 
interviewees to achieve higher marks 
although this was not statistically significant 
(f(16)=0.86, p=. 41 n.s.). Those who obtained 
higher interview grades were also more likely 
to rate the interview experience as positive 
(r=.67). 

Examination of open-text comments 
about the interview revealed the most fre¬ 
quently expressed opinions were that the 
assessment weighting (10 per cent of the 
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Table 5: Comparison of agreed interview grades for students assessed on different days of the three-day interview period. 
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Table 6: Project interview student feedback (2013 and 2014 cohorts). 
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Table 7: Project interview student feedback (2015 cohort only). 
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overall project mark) did not reflect the 
effort students had placed into preparing 
and that the interview weighting was dispro¬ 
portionate to the stress created by the assess¬ 
ment. Whilst this could be taken as 
suggesting a preference for increasing the 
assessment weighting, some students com¬ 
mented that the assessment format may have 
disadvantaged shy and nervous students, 
although none directly reported feeling that 
nerves had affected their own performance. 
Others would have preferred the interview 
to have been closer to their project report 
hand-in date. Planning is, therefore, 
required with interview assessments to 
ensure the timing and weighting of the 
assessment reflects student effort without 
creating undue anxiety. 

Discussion 

The ability to communicate research find¬ 
ings and scientific opinions to others in an 
accurate and professional manner is central 
to developing psychological literacy and an 
important employability skill for psychology 
graduates. The present study suggests oral 
interviews can be used as a reliable means of 
assessing the development of such attributes 
in undergraduate psychologists in a manner 
that is both educationally acceptable to 
students and a valid indicator of their 
general level of performance. 

For approximately six out of every 10 
students, project interview classifications 
were consistent with their overall degree 
classifications. Whilst on the surface this may 
not seem remarkable, it should be noted that 
the net contribution of the interview to 
degree classifications was small, equivalent to 
three per cent of the final year. Despite this, 
interview marks were found to correlate 
more strongly with students’ final year 
course average than they did with project 
report marks. This may suggest that the 
interview assessment is more predictive of 
the broad range of skills students develop 
across their degree and, therefore, highly 
relevant to their learning as an undergrad¬ 
uate. 


The range of topics considered during 
the interview were selected to cover each stu¬ 
dent’s ability to explain in simple terms the 
purposes and findings of a research study, 
demonstrate they understood the reasoning 
behind their work, and to show critical 
awareness of the limits of their approach 
such that its contribution could be appropri¬ 
ately framed. These were in addition to 
asking students to reflect on their own devel¬ 
opment during the research process and 
evaluating their general ability to communi¬ 
cate effectively in a professional setting. The 
assessment, therefore, provides a relatively 
efficient means by which student learning 
across several of the core skills within the 
QAA (2010) psychology benchmark and 
emphasised within recent definitions of psy¬ 
chological literacy (e.g. Cranney et al., 2011; 
Mair et al., 2013) can be evaluated. 

Final year research projects are a culmi¬ 
nating feature of undergraduate degree pro¬ 
grammes in psychology through which many 
graduate attributes are practiced. However, 
students do not always see the connection 
between what they have achieved through 
their project and their own employability 
(Healey et al., 2013). The addition of a cor¬ 
responding assessment which specifically 
encourages students to verbalise the personal 
skills they have developed through their 
project can only benefit students in future 
recruitment and selection contexts. Oral 
assessments may also help students to pre¬ 
pare for the forms of communication they 
will encounter in their future careers, more 
so than other forms of undergraduate assess¬ 
ment (Joughin, 2003). As Huxham et al. 
(2012) suggest, oral assessments act as a pow¬ 
erful tool in helping students establish a pro¬ 
fessional identity which adds to the perceived 
authenticity of this form of assessment. This 
view is consistent with the positive evaluation 
of the appropriateness and perceived future 
benefit of the project interviews reported by 
students in the present study. 

A common view expressed within the lit¬ 
erature is that oral presentations are 
resource intensive and time-consuming to 
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conduct, which can be particularly problem¬ 
atic with large class sizes (e.g. Butler & 
Wiseman, 1993; Joughin, 2003). We found 
that five or six teams of markers working 
together in a rotated pattern of pairings 
could readily accommodate cohort sizes of 
up to 180 students over a three-day period 
(10 to 12 interviews per team per day). More¬ 
over, since the interview duration was rela¬ 
tively short, time could be built into the 
schedule for mark co-ordination and the 
production of written feedback to students, 
meaning that outcomes could be communi¬ 
cated to students very quickly following the 
assessment. We estimate that the total 
person-hours for all markers would, there¬ 
fore, not far exceed the time required by one 
person to mark written assessments, marked 
at the rate of one assessment per-hour in a 
similar sized cohort. Beyond practical con¬ 
siderations of staff workload, the three-day 
interviewing period as an event in its own 
right, was found to add value to the sense of 
community within our department with pos¬ 
itive reactions being reported by both 
students and staff each year. Whilst other 
efficiency measures are possible to reduce 
time costs, such as using single examiners 
rather than interview panels (Butler & 
Wiseman, 1993) or conducting interviews 
with multiple students present (Oakley & 
Hencken, 2005) these were not felt to be 
appropriate in this context, to maintain the 
perceived fairness of the assessment, as well 
as the confidentiality of each candidate. 

Within our interview structure, supple¬ 
mentary questions were used to probe the 
boundaries of students’ knowledge. Consis¬ 
tent with Butler and Wiseman (1993), this 
appeared to be effective for gauging depth 
of understanding and for distinguishing 
between competent and exceptional candi¬ 
dates. Additional question probes were also 
particularly important in cases where 
students responded briefly to questions and 
helped avoid progressing too quickly 
through key aspects of the interview. In such 
cases, second interviewers were able to 


monitor the time spent on each interview 
theme to ensure all students had approxi¬ 
mately equivalent periods addressing topics, 
and more anxious students who might 
answer quickly were not disadvantaged. 

Some research on viva voce examinations 
suggests questions often address the recall of 
basic information rather requiring students 
to demonstrate depth of understanding (e.g. 
Davis & Karunathilake, 2005; Evans et ah, 
1966). Examiners, therefore, mistake confi¬ 
dence and articulate expression in candi¬ 
dates for enhanced subject knowledge, such 
that eloquent but weaker students may 
receive better ratings than their perform¬ 
ance warrants (Thomas et al., 1993; Torke et 
ah, 2010). In the present study, the marking 
criteria used specifically required examiners 
to separate out their assessment of the style 
and subject content of answers provided by 
candidates in an attempt to mitigate against 
such effects. Comparison of the marking 
within interview teams as well as between dif¬ 
ferent teams suggested a consistent pattern 
of grading was followed throughout the 
project interviews. In addition, no evidence 
was found to suggest that students’ interview 
performance was evaluated leniently, or was 
inconsistent with their performance else¬ 
where on their degree, which might be 
expected if confidence rather than knowl¬ 
edge was being rewarded. As McAdams and 
Robertson (2012) have also pointed out, the 
use of oral assessments in an academic pro¬ 
gramme can be justified where the mastery 
of professional skills such as using evidence 
to make a verbal case, presenting a persua¬ 
sive argument, and adapting communication 
styles to suit a particular audience are 
required within the field. It can, therefore, 
be concluded that the use of assessed project 
interviews provides a promising and reason¬ 
able means by which psychological literacy, 
and more specifically, the ability to commu¬ 
nicate scientific thinking and findings in a 
clear and appropriate manner may be evalu¬ 
ated on undergraduate programmes. 
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